After converting to the int8_bfloat16 I get this error when trying to run it in a script;

  File "D:\Scripts\bench_chat\ct2_phi3.py", line 87, in main
    results_batch = generator.generate_batch(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: expected storage to be of type float32, but is of type bfloat16

I also received the same exact message when first converting the model into bfloat16:

    results_batch = generator.generate_batch(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: expected storage to be of type float32, but is of type bfloat16

The latter is really weird because the model card says that it's originally in bfloat16...

See this if it help:

#1792

I quantized phi3.5 to int8_bfloat16 and didn't have any error like you mentioned above in inference time. Please provide more in detail how to reproduce this and which model that you used.

Sure...

The script I used to convert it is located here:

https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py

And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

It seems like you are running this script on a GPU < 8.x while it does not support compute type bfloat16. Remove this line compute_type="bfloat16" and try again. If you have any futher problem, feel free to open a new issue.

Sure...

The script I used to convert it is located here:

https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py

And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

It seems like you are running this script on a GPU < 8.x while it does not support compute type bfloat16. Remove this line compute_type="bfloat16" and try again. If you have any futher problem, feel free to open a new issue.

Sure...
The script I used to convert it is located here:
https://github.com/BBC-Esq/Ctranslate2-Converter/blob/main/Ctranslate2-Converter/convert_ctranslate2.py
And the script I used to run it is as follows:

import os
import ctranslate2
from transformers import AutoTokenizer

model_dir = r"D:\Scripts\bench_chat\models\Phi-3.5-mini-instruct-ct2-bfloat16"

def build_prompt():
    system_message = "You are a helpful AI assistant."
    user_message = "Tell me a short joke."
    
    prompt = f"""<s><|system|>
{system_message}<|end|>
<|user|>
{user_message}<|end|>
<|assistant|>
"""
    return prompt

def main():
    print(f"Loading the model: {os.path.basename(model_dir)}...")
    
    generator = ctranslate2.Generator(
        model_dir,
        device="cuda",
        compute_type="bfloat16"
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    
    prompt = build_prompt()
    tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
    
    print("Generating response...")
    results = generator.generate_batch(
        [tokens],
        include_prompt_in_result=False,
        max_batch_size=4096,
        batch_type="tokens",
        beam_size=1,
        num_hypotheses=1,
        max_length=512,
        sampling_temperature=0.00,
    )
    
    output = tokenizer.decode(results[0].sequences_ids[0])
    
    print("\nGenerated response:")
    print(output)

if __name__ == "__main__":
    main()

Are you referring to the cuda compute level? The GPU I'm running it on is an rtx 4090 so it supports compute more than 8...

Also, I just commented out this and same thing...

Do you want me to open a separate issue still? Seems redundant>